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Viithin the past fe.w years, factor analytic techniques have been 
used with increasi»:g frequency in research related to student ratings 
of courses and instructors. The method has proved useful in the sum- 
marization of such data both for purposes of description and inter- 
pretation, and for subsequent use in the development of rating instru- 
ments. The results of a factor analysis, liowever, always depend to 
some extent on such contingencies as characteristics of the data input 
and experimenter judgments at various points in the analysis, al- 
though standard answers to such questions as the number of factors to 
be extracted or rotated have been offered, such judgments should not 
be applied uncritically. The current study compares and evaluates cer- 
tain factor analytic procedures commonly employed in research related 
to students' ratings of courses and instructors. 

More specifically, the study reported here sought to determine: 

(1) the comparability of factor structures when (a) individual 
students' responses were used as the raw data input, and vjhen (b) mean 
class responses formed the data base; and 

(2) the effects of two stop^factor ing criteria on the factor 
structures obtained with each of the above data bases, that is, (a) the 
extraction of all factors with latent roots above 1.00, versus (b) the 
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continuous extraction of factors uritil about 10-J3% of the variance is 
accounted for, or until the reliable variance has been extracted. 
Data Source 

The data analyzed here are responses to 33 items on the AO-item 
Course- I nstructor Survey (CIS) General questionnaire which are spe- 
cific to ratings of courses and instructors. (The first seven items 
on the questionnaire, requesting data concerning student raters, were 
not analyzed.) Responses were coded on a 1 to 3 scale, with 1 repre- 
senting the most favorable response. The 33 items were: 

3, The instructor seemed to be sensitive to the feelings and 
needs of students. 

9. The instructor seemed well-prepared for lecture or discus- 
sion. 

10. Tlie instructor showed a scholarly grasp of the course mater- 
ial. 

11. The instructor showed confidence before the class, 

12. The instructor paced the course well. 

13. The instructor kept his lectures and class discussions fo- 
cused on the subject of the course. 

'1^. The instructor usually seemed to be aware of whether the 
class was following his presentation with understanding. 

15. The instructor used clear, relevant examples. 

16. The instructor's mannerisms or habits reduced the effective- 
ness of his teaching. 

17- The instructor's speech and lecture style contributed to his 
teaching effectiveness. 

18. The instructor made me feel free to ask questions, disagree, 
and express my ideas. 

19. The instructor was intellectually stimulating (thought-pro- 
voking, or caused me to do additional studying on my ov«/n.) 

20. The instructor showed a genuine interest in teaching the 
course. 

21. The instructor was generally accessible to students outside 
of class. 
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22. The instructor gave adequate instructions concerning assign- 
ments. 

23. The instructor commented informatively on tests and assign- 
ments . 

2k, The tests were usually graded and returned promptly. 

25. I was satisfied with the way t()e performance of students was 
evaluated in this course. 

2u. The textbooks were adequate for this course. 

17. The reference books and materials in the library were aJe*- 
quate for tiiis course. 

21). I feel that I profited from the out-of-class assignments. 

29. I feel that \ profited from the laboratory (or discussion 
section) for this course. 

30. tiefore the semester began, I thought 1 would enjoy this 
course. 

31. before the semester began, I thougtit this course v/ould be of 
value to me. 

32. At this point in time, I feel that this course will be (or 
has already been) cf value to me. 

33- The amount of outside preparation required for this course 
was : 

3^. For each hour of class, the average amount of time I spent 
on this course outside of class was about: 

35. Compared with the effort I usually put into a course, my 
effort in this course was: 

3G. 1 met with the instructor outside of class to discuss the 
course : 

37v I was absent from class: 

33. Compared with all the instructors I have had, both in high 
school and in college, this instructor was: 

33. Compared with all tiie courses I have had, both in high school 
and in college tliis course was : 

AC). Compared with what 1 expected to get from this course, I 
feel I got: 



I 

All of the 5,ol9 CIS forms completed by students in English Depart- 
ment courses during the Fall \^J7\ semester were used in the analysis* 
The survey included courses, taught by faculty and teaching assist- 
ants, both at the undergraduate and graduate levels. Participation in 
the survey, althougii urged by the University administration and by the 
Ifnglish Department, was voluntary on the part of the instructors. The 
surveys were administered under established conditions by a student mem- 
ber of each class during one of the last regularly scheduled class meet- 
ings. According to CIS procedures, the instructor is asked to leave the 
room v;hile the students complete the form, and students responded anon- 
ymously. 
'ethod 

Individual students* responses to the 33 items were i ntercorrel ated 
and a principal components factor analysis was performed using Computer 
Program FACTOR (Veldman, 1367, 1971)* Since not all students answered all 
questions, the missing-data option available witli FACTOi^ was utilized, 
when individual students' ratings were analy2:ed. The correlation matrix 
input to the factoring routine therefore consisted of coefficients com- 
puted only for valid score pairs, or for students responding to both items 
of the pair being correlatec^. 

The principal components analysis was then performed on this matrix 
of item i ntercorrelat i ons , with the initial valu^^s in the principal diag- 
onal being set equal to 1.00. The initial analysis was performed for the 
extraction of only the principal components w i th correspond i ng latent roots 
greater than 1.00, as recommended by Kaiser & Caffrey (19(^5) • 

The same analyses were then performed using the average class responses 
of the 23h participating classes on each item, and the solutions for the 
two data bases were then compared by re-rotation using Program RELATE 



-5- 

(Veldman, 1367, 196.3, 1:371). First, a factor cosine matrix representing 
the relative proximity of the factors defined from the two data bases was 
obtained. Then, the Vari max /factors computed from the class mean responses 
were re-rotated to conform as closely as possible to the target solution 
based on individual students* responses. Finally, a variable cosine matrix 
was computed which describes the rel at ionsh i p between individual item vec- 
tors in the target solution and the re-rotated solution based on class mean 
data. 

In order to test the tigenroots of 1.00 criterion, which has become 
something of a ^'blind" standard in much factor vjork, subsequent factor anal 
yses were performed utilizing an approach in which factors are extracted 
until a desired portion of the total variance has been accounted for. 
This technique has been discussed by Guertin and iJailey (11)70), Harinan 
. OS'oO) y and Horst (1968), and appeared to offer a viable alternative to the 
Eigenroots criterion, particularly in consideration of the relatively high 
reliability of instructor rating forms. In the* current study, a solution 
vjas sought which would account for about 70 to 75^ of the variance, a 
figure based on reliabilities obtained for this and similar rating scales. 
Analyses extracting increasing numbers of factors beyond six were per-- 
formed for both data bases, then, until the percentage of variance cri- 
terion had been reached, [Results of the factor solutions obtained using 
each criterion for stopping factoring were then compared. 

ResuHs 

When all latent roots above 1.00 were extracted, the factor analyses 
for both individual item responses and mean class responses yielded six 
factors. The Vartmax loadings for these solutions are shown in Tables 1 
and 2. There were, however, some differences discernible between the two 
sets of loadings. The diagonal elements in the cosine matrix shown in 



Table 3 Indicate that, while Factors II, III, and IV are quite close for 
the two solutions. Factors V and VI, and to a lesser degree, Factor i, * 
seen) to differ for the two data bases. By inspecting the primary loadings" 
(underlined in Tables. 1 and 2) for the two solutions, it can be seen that 
Factors V and VI seem simply to liave occurred in reverse order for the 
two solutions. These differences virtually disappeared, however, when 
the solution based on class means was re-rotated towards the solution 
based on individual students' responses. Table ^ shows the re-rotated 
loadings for the jnean class responses, and Table 5 gives the item vector 
cosines, indicating the relationship betv;een the 33 item vectors based on 
Individual responses and the re-rotated vectors based on mean class re- 
sponses* Table 6 lists the pr t mary load i ngs for each of these solutions, 

These data, then, indicate that the two data bases produce comparable 
solutions when the Eigenroot of 1.00 is used as tiie criterion for stop-, 
ping factoring. The six factors obtained were identified as; ^^l'. Gen- 
eral Course- Instructor Effectiveness, reflected by items related both to 
aspects of the instructor's teaching performance and competence and to 
general evaluation of the course conterit and methods; It. Student 
effort, reflecting the students' estimates of t i me , .effort , and prepara- 
tion required for the course; III. -txpectations for the Course, includ- 
ing primary loadings for two items involving students' expectations and 
for an item cnncerning textbooks; IV. Direct Contact with Instructor, 
which included primary loadings for two items related to instructor's 
accessibility and to actual conferences with the instructor; V, Instruc- 
tional Style, reflecting specific aspects of the Instructor's preparation, 
performance, and demeanor; and VI.. Evaluation and Planning, which picked 
up items related to tests and grading, and course pacing, as well the 
item concerning reference books and library materials. 
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Although the factors obtained for the two data bases were similar, 

these six factors accounted for only 55% of the variance of individual 
responses > in contrast to 73^i of the variance of mean class responses. 
In terms of the percentage of variance, criterion for stopping factor- 
ing which was proposed here, there seemed to be little purpose in ex- 
tracting more than ssx factors v;hen mean responses were used. When 
factoring was extended, twelve factors were required to account for 
70.35?o of the variance of individual responses. This solution, more- 
over, appeared to add some clearly interpretable factors which further 
clarified the dimensions underlying ratings of courses and instructors. 
When, for purposes of comparison, tv^elve factors were obtained for the 
mean class data, SS% of the variance was extracted, but there were indi- 
cations of increasing disintegration of the factor structure. As Table 
b shows, six of the factors (V, VI. VMM, fX, X, and X I ) so obtai ned 
represented the factor of primary loading for only one item, and one 
factor (aII) had no primary loadings* 

Table 7 shows the increases in latent roots and percentages of 
variance accounted for as increasing numbers of factors (from 1 to 12) 
are extracted for each data base. Although the percentages themselves 
are higher at each step for mean class responses, it should be noted 
that the gain in percentage of variance accounted for from six to twelve 
factors is somewhat higher for individual responses. 

Inspection of loadings for the twelve factors based on individual 
students' responses resulted in the factors being labeled as follows: 
1. General Course Effect i veness ; II. Student Effort; III. Expecta- 
tions for Course; tV. Direct Contact with Instructor; V. Instruction- 
al Style; Vf. Texts, References, and Materials; VN, Class Attendance; 
VIII. Instructor Empathy; IX. Instructor Mannerisms; X. Evaluation; 
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Xf. Organization; XII. Value of Lab or Di scuss ion -Sess Ions. 
Conclusions 

— » — t 

Individual students' responses and mean class responses seem to pro- 
vide comparable factor structures in terms of the large, general factors. 
When more specific and meaningful factors are sought, howevei', it is 
recommended that factoring be extended beyond the Eigenroot of 1.00 cri- 
terion, extract ing factors until the variance accounted for is equal to 
the percentage of reliable test variance, or until the percentage of var- 
iance accounted for is increased as much as possible without producing 
factor f i ss ion. 

The data presented here call attention to inappropriate use of ■ 
the Eigenroot criterion in factor work, a practice which has probably 
grown out of a theoretical focus ov^determini ng basic laws of behavior, 
rather than a practical emphasis on providing information that will be 
useful to college instructors and administrators, ^/hen researchers nave 
sought only the general, broad dimensions that. occur in CIS ratings, 
instructors and administrators have consequently concluded that only 
these larger factors are needed in describing the evaluation of courses 
and instructors. In reality, however, administrative decision making 
and the provision of comprehensive feedback to instructors may demand 
that more specific factors be identified. tn such cases, the proportion 
of variance criterion for stopping factoring may reveal information which 
would be lost- were the Eigcnroot criterion appl t ed wi thout regard to 
variance extracted. The question of when to stop factoring, then, seems 
to'be best answered in terms of whether general laws of behavior are 
sought, or basic dimensions for specific practical applications. 

Vihen the proportion of variance criterion is used, smaller factors 
can be identified and expanded, which might otherwise have appeared as 
O .-/^HMnH.'t^nt items loading on the more general factors. The practice of 

ERIC 
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extracting only these 1 arger factors , moreover, itas sometimes led to 
•speculation that considerable response bias exists in course and in- 
structor ratings, often resulting in distrust of such ratings by in-- 
structors. Misapplied factor methodology, then, can result both in 
obscuring information and in limiting the use of CIS results. The 
current paper offers soma guidelines which should facilitate more ap- 
propriate applications of factor analytic techniques within this area. 
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. 1636 


.1235 


.0272 


.0066 


.7503 


.2145 


10 


.2188 


.1173 


.0951 


-.0199 


.7355 


.1033 


n 




.0739 


.0293 


.0092 


.7269 


. 1216 


12 


.2270 


-.0203 


-.0132 


-.0141 


.4117 


. 4547 


13 


.0^38 


.oi;63 


.0981 


-.0433 


.4932 


.4069 


]k 


.31)69 


-.0843 


-.0340 


.2336 


.4420 


.3219 


15 


.3313 
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. lOiJfj 


.5469 . 
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16 
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.0328 
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17 
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-.0110 


.0326 


.1291 


.511 1 
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.1111 
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21 
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-3516 


22 


.2030 
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23 
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-.0777 
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2k 
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. 1097 
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. 1242 
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.6250 


25 


.5056 
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-.0247 


. 0779 


.1949 


.4597 


26 


.3249 


.0325 


.3492 


-.3063 


. 1047 


.3182 


27 


. 1848 


.0524 


.2602 


-.1409 


-.0009 


.4331 


28 


.6117 


. 1932 


.2317 


-.0001 


. 1068 


.3030 


29 


.4684 


• 1733 


. 309 1 


-.0209 


.0332 


.2419 


30 


.0247 


.0433 


.8605 


.0006 


.0334 
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31 


. 1 200 


.0997 
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.3661 


-.0013 
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32 


.7180 
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-.0488 
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.1223 


.0314 
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.1221 


.0863 
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.0478 


35 
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.7513 


.0444 


.1099 


.1430 


.0104 


36 


.0713 


.2741 


.0243 


.7074 


-.0415 


-.0095 


37 


.2096 


. 4096 


-.2185 
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-.1494 


.1195 


38 


.6307 


.0952 


-.0126 


.1359 
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39 


.7019 
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.3598 
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.715s 


. 1654 


-.2358 


.0032 


.3047 


. 1461 



TAuLu 2 



VARIIIAX FACTOR LOADI.iGS FOR A.iALYSIS OF 
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.0075 


•-.0550 
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.0129 
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.1315 
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-.2390 


-.1334 


.0516 


-.0079 


.0902 


13 


.7609 - 


.1185 


-.4546 


.1933 


-.0004 


.0093 


20 


.6403 


.1493 


-.4145 


.3466 


.0740 


-.0917 


21 


. 1-138 


.2484 


-.0334 


. 7965 


. 1999 


-.0502 


22 


-3733 


-.2817 


-.1235 


.3079 


.5354 


-.0142 


23 


.4631 


-.1504 


-.0026 


.5544 


.4398 
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2k 
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.2712 


.5568 
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-.7116 
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-.043Li 
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28 


.5152 


.2395 


-.5253 


.2227 
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.2344 


29 


.2338 
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.1240 


.0813 


30 
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-.2300 
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. 1539 


36 
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.5827 


. 1034 


.5572 


-.1424 
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37 


.0348 


.3894 


.0734 


.0570 


.0134 


.7125 


33 


.8/08 


. 1302 


-.255C 
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.1128 


.0913 


39 


.7239 


.2344 


-.5013 
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.2368 
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\ 


.635;? 


-.1063 


-.3653 


.2618 


-.3455 


.5129 


2 


-.0164 


.9371 


-.0464 


-.1521 


.0654 


.3032 


3 


-.1874 


. 1005 


- . 8739 


.0494 


-.0350 


-.4227 




-.0013 


.2535 


.2276 


. 8844 


-.0375 


- . 2996 


5 


.7458 


.1249 


.0736 


-.1854 


.3641 


-.5057 


6 


-.OG73 


-.1340 




.2933 


.3562 


.3475 



Cosines on the diagonal are directly interpretable as correUtion 
coefficients between the two sets of factor loadings. 
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. / 


.0167 
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-.1055 
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. «. 1 ^ (J 
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. *-\^\^ J 


26 
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1 0 ] a 
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33 




.1722 


.0465 


. 1 192 


.6124 


. 1 50!i 


39 


.6664 


.2482 


.2304 


.0276 


.4980 


.1913 


40 


■ .6041 


.1634 


-.1303 


.0310 


.5359 


.2833 



TaULE 5 



CJSI,JL3 UlTUuEiJ ITEpI VlICTOl^vS FOR SIX FACTORS 
3ASEJ 0,.J hit) IVI DUAL A,JD HEA.J CLASS RLSPU,iSl£S 



I tern Ao. 



10 
11 
12 

(3 

]k 

15 
16 

17 
18 
11) 
20 
21 
22 

23 
21* 



Item Vector 
Cos i ne 



.JjO'd 
.9763 

.9366 
.■j2^2 
.9601 
.9600 
.9079 

.9639 

.9389 

.9662 

.93J6 

.37hO 

.967V 

.343^ 



Item ;io. 



2'S 
26 

27 

23 



29 



30 
31 

32 
33 
34 
35 
36 
37 
30 
39 
^0 



Item Vector 
Cos i ne 



.970^1 



.9517 



.3332 
.9736 
.9390 
.9535 
.9418 
.9653 
.906^* 
.9856 

.95A3 
.99^*6 
.9928 
.9694 
.9332 



ERIC 



TAliLE 6 



COMPARISON OF PRIMARY LOADINGS FOR FACTOR SOLUTIONS OF 
SIX AND TWELVt FACTORS DEillVIIO FROM TWO UATA BASES 



'six-factor SOLUTIONS 



Individual Class 
Responses Averages" 

Factor I 



32 


• 7130 


17 


.7014 


i»0 


.7150 


19 




39 


.7019 


38 


.6360 


30 


.6307 


39 


.666^ 


19 


.6303 


32 




20 


.6117 


3 


.G25I 


17 


.5^12 


23 


,6185 


25 


.5056 


40 


.6041 


8 


.5042 


14 


.6027 


10 


.A752 


20 


.5609 


23 


.i»684 


18 


.5531 






27 


.5385 




Factor 


1 1 




3'* 


• 7591 


33 


. 3O5O 


35 


.7513 


34 


.8052 


33 


• 7^91 


35 


.8113 


37 


. '1O96 


37 


.5637 




Factor 


1 1 1 






. 8605 


30 


.8560 


31 


-3162 


31 


.7248 


2& 


.3't92 


26 


. 5403 






29 


.4383 




Factor 


IV 




36 


.7074 


21 


.7564 


21 


.6575 


36 


.6520 




Factor 


V 




'J 


.750G 


9 


.8033 


10 


.7355 


10 


. 7478 


1 1 


.7269 


11 


. 7363 


15 


. 5'469 


15 


.6187 


13 


.A9G2 


16 


.5227 


20 


.4895 






\k 


.4420 






10 


.4268 







r/tLVt-FACTUR SOLUTIOiJS 



Individual Class 
llesponses Averages 

Factor I 



40 


.7430 


1 1 


. 8754 


32 


.7346 


33 


.8433 


39 


.7302 


10 


.8403 


28 


.5978 


17 


.7781 


38 


.5580 


9 


.7763 


19 


.4337 


15 


.7575 






40 


.7506 






19 


.7374 






39 


.7095 






20 


.6518 






32 


.6332 






14 


.6004 






23 


.4311 




Factor 


1 1 




33 


.0257 


33 


.8257 


34 


. 8039 


34 


.3039 


35 


. 7506 


35 


.7506 




Factor 


i 1 1 




30 


.3716 


30 


.9058 


31 


.870] 


31 


.8355 






29 


.5941 






26 


.5420 




Factor 


IV 




36 


.3767 


36 


.8653 


21 


.6753 


21 


.7676 




Factor 


V 




10 


.7642 


13 


.0834 


3 


.7452 






1 1 


.7305 






17 


.4774 






20 


.4721 






15 


. 4649 







SIX-FACTOR 50LUTICMS 



TV/ELVE-FACTOR SQLUTiO>4S 



Individual Class 
Responses Averages 

Factor Vl 



Ik 


.C250 


22 


. 60A9 


23 


.5320 


12 


.6012 


22 


• iiiiO? 


m 


.568i» 


12 


Mkl 


23 




27 


.4321 


13 


.5101 






25 


.4965 



" Ks- rotated Solution. 



Individual Class 
Responses Averages 

Factor Vl 

27 .8813 37 .9032 
26 .5749 

Factor Vli 

37 .9464 22 .3133 

1 ;3 . 7404 

23 .7234 

25 .6306 

8 .6243 

Factor VIII 

18 :7734 27 .9040 

8 .6475 

14 .5933 

23 .5115 
25 .4637 

Factor IX 
i6 .3900 24 .9448 
Factor X 

24 .8365 12 .7436 

Factor XI 

13 .7075 16 .6584 
22 ,5461 
12 .5161 

Factor XI I 

29 .8096 



TABLE 7 





•Variance 


Accounted for by 


Successive Principal 


Axes 




L)ata Based on Individual 


Student 


Responses 




Factors 


Size of 
Latent Root 


Difference From 
Previous Latent Root 


Cumulative 
Latent Roots 


Cumulative. Z of 
Total Variance 
Accounted for 


1 


3.97 






3-97 


30.21 


II 


2.56 


7. '♦I 




12.53 


37. 8A 


1 1 1 


1.79 


.77 




14.32 


'»3.27 


ly 


1 .50 


.29 




15.82 


A7.82 


V 


1.27 


.23 




17.09 


51.63 


VI 


1.13 


.U 




13.22 


55.09 


Vli 


.93 


.20 




19.15 


57.90 


VIII 


.9A 


.01 




20.09 


GO. 7k 


IX 


-00 


.U 




20.89 


63.16 


X 


.8i» 


.Ok 




21.73 


65.71 


XI 


.31 


.03 




22. S'* 


68.17 


XII 


.72 


.03 




23.26 


70.35 



Data iiased on Average Class Responses 



ERIC 



1 


13.69 




13.63 


Al .50 


11 


3.71 


9-93 


17. AO 


52.73 


Ml 


2. A3 


1 .23 


19.83 


60.09 


IV 


1 .32 


.61 


21.65 


65.60 


V"', 


1.3A 


.^♦8 


22.99 


69.79 


VI 


1 .Ok 


.30 


2k. 03 


72.3k 


VII 


.86 


.18 


2k. 83 


75.55 


VIII 


.76 


.10 


25.65 


76. 8A 


IX 


.77 


.01 


2G.k5 


79.18 


X 


.72 ' 


.05 


27.17 


82.37 


XI 


. .57 


.15 


27.7k 


84.08 


XII 


.59 


.02 


28.33 


35.35 
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